Skipping refinement based on patch similarity in bilinear interpolation based decoder-side motion vector refinement

ABSTRACT

Methods and apparatuses of determining an alignment level between motion compensated reference patches for reducing motion vector refinement steps are provided. According to one method, obtaining, by a decoder, motion compensated interpolated samples based on sub-pixel accurate merge motion vectors from a bilinear motion compensated interpolation; computing, by the decoder, a sum of absolute differences (SAD) between two motion compensated reference patches using a subset of the motion compensated interpolated samples; determining, by the decoder, whether the SAD is less than a coding unit (CU) size-dependent threshold value; when the SAD is less than the CU size-dependent threshold value: skipping remaining decoder-side motion vector refinement (DMVR) process steps; and
         performing final motion compensation; and when the SAD is not less than the CU size-dependent threshold value: performing the remaining DMVR process steps; and performing the final motion compensation.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of International Application No.PCT/CN2019/106308, filed on Sep. 18, 2019, which claims priority toIndian Provisional Patent Application No. IN201831035309, filed on Sep.19, 2018. The disclosures of the aforementioned patent applications arehereby incorporated by reference in their entireties.

TECHNICAL FIELD

The present disclosure relates to motion compensation for video codingusing decoder-side motion vector refinement. In particular, embodimentspresented herein relate to methods and apparatuses for verifying analignment level between motion compensated reference patches andskipping refinement when a difference between the two motion compensatedreference patches is less than a coding unit size-dependent thresholdvalue.

BACKGROUND OF THE INVENTION

In video compression, inter prediction is a process of usingreconstructed samples of previously decoded reference pictures byspecifying motion vectors relative to a current block. These motionvectors can be coded as a prediction residual by using spatial ortemporal motion vector predictors. The motion vectors can be atsub-pixel accuracy. In order to derive the sub-pixel accurate pixelvalues in the reference frames (pictures) from the reconstructed integerposition values, an interpolation filter is applied. Bi-predictionrefers to a process where the prediction for the current block isderived as a weighted combination of two prediction blocks derived usingtwo motion vectors from two reference picture areas. In this case, inaddition to the motion vectors, the reference indices for the referencepictures from which the two prediction blocks are derived also need tobe coded. The motion vectors for the current block can also be derivedthrough a merge process where a spatial neighbor's motion vectors andreference indices are inherited without coding any motion vectorresiduals. In addition to spatial neighbors, motion vectors ofpreviously coded reference frames are also stored and used as temporalmerge options with appropriate scaling of the motion vectors to takecare of the distance to the reference frames relative to the distance tothe reference frames for the current block.

FIG. 1 shows a template matching based decoder-side motion vectorderivation, where the template of the current block is matched with thereference template in a reference picture. Referring to FIG. 1, templatematching is employed to derive motion information of the current codingunit (CU) by finding the closest match between a template (top and/orleft neighboring blocks of the current CU) in the current picture(denoted “Cur Pic”) and a block having the same size as that of thetemplate in a reference picture (denoted “Ref0”).

FIG. 2 shows a bilateral matching based decoder-side motion vectorderivation, where a current block is predicted using two referenceblocks along a motion trajectory. Referring to FIG. 2, motioninformation of the current block (denoted “Cur block”) is derived basedon two reference pictures Ref0 and Ref1. The motion information of thecurrent block Cur block is derived by finding the best match between twoblocks associated with the motion vectors MV0 and MV1 along a motiontrajectory in the reference pictures Ref0 and Ref1. When the motiontrajectory is a straight line, the motion vector MV0 associated withreference picture Ref0 and the motion vector MV1 associated withreference picture Ref1 are proportional to the temporal distances TD0and TD1 between the current picture and the respective referencepictures Ref0 and Ref1.

Several methods have been proposed for performing a decoder-side motionvector refinement or derivation so that the motion vector residualcoding bits can be further reduced. One class of methods, calledtemplate matching (TM) methods, use an L-shaped region adjoining thecurrent block (as shown in FIG. 1) that has already been reconstructed,referred to as the template, and identifies a best matching L-shapedregion (using cost functions such as a sum of absolute differences ormean-removed sum of absolute differences) in each reference frame usinga plurality of suitably scaled spatial and temporal motion vectorcandidates. Then, centered on the best matching candidate, furtherrefinement is performed within a certain refinement distance around thatcenter. On the encoder side, rate distortion optimized cost is computedto decide between uni-prediction (i.e., prediction using the bestmatching reference) and bi-prediction (i.e., prediction derived byaveraging the top two best matching references).

Another class of methods, called bilateral matching (BM) methods derivemotion information of the current coding unit (CU) by finding theclosest match between two blocks along the motion trajectory of thecurrent CU in two different reference pictures. This is shown in FIG. 2.Under the assumption of continuous motion trajectory, the motion vectorsMV0 and MV1 pointing to the two reference blocks shall be proportionalto the temporal distances, i.e., TD0 and TD1, between the currentpicture and the two reference pictures. When the current picture istemporally between the two reference pictures and the temporal distancefrom the current picture to the two reference pictures is the same, thebilateral matching becomes mirror based bi-directional motion vector(MV).

In the bilateral matching merge mode, bi-prediction is always appliedsince the motion information of a CU is derived based on the closestmatch between two blocks along the motion trajectory of the current CUin two different reference pictures.

Explicit merge mode to indicate template matching merge or bilateralmatching merge can be signaled to differentiate these modes from adefault merge mode that does not require any decoder-side motion vectorderivation.

In the bilateral matching mode, the temporal distances are ignored andbilateral matching is performed with equal and opposite motion vectorsin the past and future reference frames respectively.

In some cases, no merge index is signaled while in other cases, tosimplify the decoder complexity of performing multiple motioncompensations, an explicit merge index is signaled.

In a variant of bilateral matching mode called decoder-side motionvector refinement (DMVR) method, a bilaterally averaged template isfirst created using the prediction blocks in reference lists L0 and L1obtained from explicitly signaled merge index and bilateral matching isperformed against this template. The template is updated if there is anymovement. Also, in some cases, the refinement is performed in onereference and the motion vector in the other reference is obtainedthrough mirroring of this refined motion vector. The refinementalternates between the two references until either the center positionhas the least error or the maximum number of iterations is reached.

In some refinement processes, a CU level refinement is first performed.Then a sub-CU level multi-candidate evaluation is performed along withthe CU-level refined MVs as candidates. In other refinement processes,each sub-CU can perform its own refinement with respect to the bestmatching candidate.

Given the implicit decoder-side derivation or refinement process, theencoder needs to perform these steps in exactly the same manner as thedecoder in order for the encoder-side reconstruction to match with thedecoder-side reconstruction.

Only luma samples are typically used during the decoder side motionvector refinement or derivation process. However, chrominance is alsomotion compensated using the final refinement motion vectors (suitablyscaled to account for any chroma downsampling) as used for luma motioncompensation.

Since the merge motion vectors are sub-pixel accurate, the refinement istypically performed initially at integer distance from the respectivesub-pixel accurate merge motion vectors in each reference. Since thenormative motion compensation is computationally expensive, a bilinearinterpolation method is commonly used to create the interpolated samplevalues required for performing the integer distance refinement. Otherprior art techniques made proposal of using cost function valuesevaluated at integer distance positions and the cost function value atthe position with the lowest cost when the refinement ends to obtain aparametric error surface using which sub-pixel delta motion vectorcorrections are estimated. Once the final integer distance plussub-pixel distance based motion vector corrections in each reference isobtained, final normative motion compensation is performed.

BRIEF SUMMARY OF THE INVENTION

Embodiments presented herein relate to methods and apparatuses fordetermining the level of alignment between motion compensated referencepatches against a predetermined coding unit size dependent thresholdvalue. By determining the alignment level between motion compensatedreference patches against the predetermined coding unit size dependentthreshold value, computational operations of the decoder-side motionvector refinement iterations can be saved, thereby reducing powerconsumption in a video decoder.

Methods and apparatuses of determining an alignment level between motioncompensated reference patches for reducing motion vector refinementsteps are provided. According to one method, a video decoder determinesthe alignment level by rounding merge motion vectors, calculates the sumof absolute differences (SAD) between the two motion compensatedreference patches by performing simple motion compensation using therounded motion vectors, and determines whether the SAD is less than a CUsize-dependent threshold value. When the SAD is less than the CUsize-dependent threshold value, the decoder skips the remainingdecoder-side motion vector process steps, and performs final motioncompensation using the unrounded merge motion vectors. In oneembodiment, the unrounded merge motion vectors may be rounded to anearest integer sample position in the horizontal direction and in thevertical direction, and the final motion compensation includesperforming an averaging of the integer position samples. In anotherembodiment, the unrounded merge motion vectors may be rounded to thenearest half-pixel sample position in the horizontal direction and inthe vertical direction to obtain half-pixel position samples, and thefinal motion compensation includes performing an averaging of thehalf-pixel position samples. The method is advantageous in that wheneverthe SAD is less than a predetermined threshold value, the video decoderdetermines that the alignment level between motion compensated referencepatches is acceptable and the video decider can skip the motion vectorrefinement processes such as bilinear interpolation, mean valuedetermination, refinement cost function evaluations, and other processsteps to save clock cycles and reduce power consumption. In anothermethod, the video decoder calculates the SAD between the two motioncompensated reference patches using a subset of samples from a bilinearmotion compensated interpolation using sub-pixel accurate merge motionvectors. The SAD is then compared with a coding unit (CU) size-dependentthreshold value. The bit depth of the coding unit size-dependentthreshold value can be adjusted according to the bit depth of theinterpolated samples. By computing only the SAD using a subset ofinterpolated samples, the method advantageously requires feweroperations (clock cycles). In yet another method, the decoder calculatesa mean-reduced SAD (MR-SAD) between motion compensated interpolatedsamples at center positions of respective motion compensated referencepatches and determines whether the MR-SAD is less than a CUsize-dependent threshold value. In this method, the MR-SAD of the centerpositions of the two motion compensated reference patches is computedwhere the refinement begins. When the MR-SAD for the center positions isless than the CU size-dependent threshold value, the video decoder skipsthe remaining of the decoder-side motion vector refinement process. Aninter prediction method, comprising: obtaining a first motioncompensated reference patch and a second motion compensated referencepatch based on initial motion vectors of Merge Mode; computing a patchdifference value between the first motion compensated reference patchand the second motion compensated reference patch; obtaining aprediction value of a current block based on the initial motion vectorsin the event that the patch difference value is less than a threshold,wherein the threshold is determined based on a size of the currentblock; and refining the initial motion vectors to obtain the perditionvalue of the current block in the event that the patch difference valueis greater than or equal to the threshold. In a feasible implementation,the initial motion vectors are derived from motion vectors of aneighboring block of the current block. In a feasible implementation,the initial motion vectors comprises a first motion vector and a secondmotion vector, wherein the first motion compensated reference patch isobtained according to the first motion vector, and the second motioncompensated reference patch is obtained according to the second motionvector. In a feasible implementation, the patch difference value is aSAD value or a mean-reduced SAD value. In a feasible implementation,computing the patch difference value between the first motioncompensated reference patch and the second motion compensated referencepatch, comprising: computing the patch difference value between a subsetsamples of the first motion compensated reference patch and acorresponding subset samples of the second motion compensated referencepatch. In a feasible implementation, after computing the patchdifference value between the first motion compensated reference patchand the second motion compensated reference patch, further comprising:performing a comparison based on the patch difference value and thethreshold. In a feasible implementation, before performing thecomparison based on the patch difference value and the threshold,further comprising: adjusting the threshold after determining thethreshold according to the size of the current block. In a feasibleimplementation, in the event that the patch difference value is lessthan the threshold, before obtaining the prediction value of the currentblock based on the initial motion vectors, further comprising:determining decoder-side motion vector refinement (DMVR) is skipped. Ina feasible implementation, in the event that the patch difference valueis less than the threshold, obtaining the prediction value of thecurrent block using motion vectors that are the same as the initialmotion vectors. An inter prediction apparatus, comprising: anon-transitory memory having processor-executable instructions storedthereon; and a processor, coupled to the memory, configured to executethe processor-executable instructions to facilitate the method accordingto any one of feasible implementation of the above inter predictionmethod. These novel techniques save substantial iterations of the motionvector refinement, thereby reducing the number of clock cycles and powerconsumption of the video decoder.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a template matching based decoder-side motion vectorderivation, where the template of the current block is matched with thereference template in a reference picture.

FIG. 2 shows a bilateral matching based decoder-side motion vectorderivation, where a current block is predicted using two referenceblocks along a motion trajectory.

FIG. 3 shows an example of a bilateral template matching baseddecoder-side motion vector refinement (DMVR), where a template isgenerated by using the bi-prediction from the prediction blocks referredby the initial motion vectors MV0 and MV1 and the bilateral templatematching to find the best matched blocks referred by the updated motionvectors MV0′ and MV1′.

FIG. 4 is a simplified flowchart illustrating a method of determining analignment level between motion compensated reference patches accordingto an embodiment presented herein, where the conditional skipping of therefinement process steps is based on a rounded integer position-basedcost function.

FIG. 5 shows a simplified flowchart of a method using decoder-sideverification, checking or determination of an alignment level betweenmotion compensated patches according to an embodiment presented herein,where the conditional skipping of the refinement process steps is basedon a sum of absolute differences between two motion compensatedreference patches using a subset of motion compensated interpolatedsamples.

FIG. 6 shows a simplified flowchart of a method using decoder-sideverification, checking or determination of an alignment level betweenmotion compensated patches according to an embodiment presented herein,where the conditional skipping of the refinement process steps is basedon a mean-reduced sum of absolute differences between motion compensatedinterpolated samples at center positions of respective motioncompensated reference patches.

FIG. 7 is a block diagram showing an example structure of a contentsupply system 3100 which realizes a content delivery service.

FIG. 8 is a block diagram showing a structure of an example of aterminal device.

DETAILED DESCRIPTION OF THE INVENTION

The present disclosure relates to versatile video coding standardizationwhich was earlier pursued as a Joint Exploratory Model (JEM) withinJoint Video Exploration Team which is a joint work between Q16 of VCEGand MPEG (SC29/WG11). Document JVET-G1001 and other Huawei prior artrelating to decoder-side motion vector refinement and decoder-sidemotion vector derivation can be used to get a list of contributiondocuments and patents related to the disclosure herein.

As explained above, the bilateral matching process tries to improve thelevel of alignment between the motion compensated blocks in the tworeferences involved in bi-prediction. In order to avoid coding an extraflag, the preferred option is for all merge mode inter-coded codingunits to be eligible for decoder-side motion vector refinement. However,it is noticed that quite a good number of coding units have very goodalignment already at the merge mode motion vectors and do not have muchroom for improvement through refinement. Since refinement involvesperforming cost function evaluations at multiple positions within therefinement range, avoiding these operations when there is no possibleimprovement that can result in considerable reduction of average decodercomplexity which can translate to attributes such as better battery lifeor lower consumption of energy. Some prior art methods have proposedskipping the refinement if the level of alignment between the twonormative motion compensated patches obtained from the merge motionvectors in bi-predictive merge mode coding units is less than apre-determined threshold for each coding-unit size. However, given thefact that bilinear interpolation is commonly employed to create thesamples used for performing refinement, prior art would still require anormative motion compensation for checking whether refinement can beskipped, bilinear interpolation for performing the refinement, and finalnormative motion compensation with the sub-pixel accurate refinementmotion vectors. Since the normative 8-tap interpolation filters arequite computational expensive, performing the interpolation filteringtwice makes the worst-case complexity higher than without the test forskipping the refinement. Also, from a hardware timing point of view, thesum of absolute difference evaluation adds a dependent stage thatreduces the clocks available for refinement.

Hence there is a need to harmonize the concept of checking to skip therefinement iterations without increasing the interpolation cycles (whencompared to the bilinear interpolation based refinement with final exactmotion compensation using refined motion vectors) and without worseningthe worst-case timing constraints for hardware implementations.

Motion vector (MV) refinement is a pattern based MV search with thecriterion of bilateral matching cost or template matching cost. In thecurrent development, two search patterns are supported—an unrestrictedcenter-biased diamond search (UCBDS) and an adaptive cross search for MVrefinement at the CU level and sub-CU level, respectively. For both CUand sub-CU level MV refinement, the MV is directly searched at quarterluma sample MV accuracy, and this is followed by one-eighth luma sampleMV refinement. The search range of MV refinement for the CU and sub-CUstep are set equal to 8 luma samples.

In bi-prediction operation, for the prediction of one block region, twoprediction blocks, formed using a MV of list0 and a MV of list1,respectively, are combined to form a single prediction signal. In thedecoder-side motion vector refinement (DMVR) method, the two motionvectors of the bi-prediction are further refined by a bilateral templatematching process. The bilateral template matching applied in the decoderto perform a distortion-based search between a bilateral template andthe reconstruction samples in the reference pictures in order to obtaina refined MV without transmission of additional motion information.

FIG. 3 shows an example of a bilateral template matching baseddecoder-side motion vector refinement (DMVR), where a template isgenerated by using the bi-prediction from the prediction blocks referredby the initial MV0 and MV1 and the bilateral template matching to findthe best matched blocks referred by the updated MV0′ and MV1′. First,the best match MV is set for the current block in a merge mode to be thecandidate MV0 in list0. Similarly, the best match MV is set for thecurrent block (in the merge mode) to be the candidate MV in list1. Thereference pixels are then averaged to form a template. Next, using thetemplate, the surrounding regions of the candidate MVs of the first andsecond reference pictures are searched, and the MV with the lowest costis determined to be the final MV. The cost value may be calculated usinga sum of differences between each pixel in the template and each pixelin the regions searched. Referring to FIG. 3, in step 1, a bilateraltemplate is generated from the prediction blocks referred to by theinitial motion vectors MV0 and MV1 in the reference blocks in list0 andlist1, respectively.

In step 2, a bilateral matching is to find the best matched blocksreferred by the updated motion vectors MV0′ and MV1′. The templatematching operation includes calculating cost measures between thegenerated template and the sample region (around the initial predictionblock) in the reference picture. For each of the two reference pictures,the MV that yields the minimum template cost is considered as theupdated MV of that list to replace the original one. In the currentdevelopment, nine MV candidates are searched for each list. The nine MVcandidates include the original MV and 8 surrounding MVs with one lumasample offset to the original MV in either the horizontal or verticaldirection, or both. Finally, the two new MVs, i.e., MV0′ and MV1′, areused for generating the final bi-prediction results. A sum of absolutedifferences (SAD) is used as the cost measure. DMVR is applied for themerge mode of bi-prediction with one MV from a reference picture in thepast and another from a reference picture in the future, without thetransmission of additional syntax elements.

In various embodiments, the difference between two motion compensatedreference patches is compared or measured against a pre-determinedcoding unit size dependent threshold value, and the refinement isskipped based on the comparison or measurement result. The differencebetween two motion compensated reference patches is performed eitherusing pre-interpolation samples or using a subset of the bilinearinterpolated samples. Also, in some embodiments, a mean-removed sum ofabsolute differences (MR-SAD) of the patches may be used as the errormetric for refinement, simple sum of absolute differences of the patchesmay be used as the metric on which the skipping check is performed inorder to allow saving of the mean computations on the average and alsoto help with the worst-case timing requirements. The pre-determinedthreshold value is adapted to the lower bit-depths that may be employedduring or after interpolation to reduce the buffer size and processingrequirements.

Given that decoder-side motion vector refinement/derivation is anormative aspect of a coding system, the encoder will also have toperform the same error surface technique in order to not have any driftbetween the encoder's reconstruction and the decoder's reconstruction.Hence, all aspects of all embodiments are applicable to both encodingand decoding systems. Embodiments provided below are merely someexamples for understanding the present disclosure and thus should notunderstood as being intended to be limiting.

Embodiment 1

In this embodiment, the merge motion vectors are rounded to the nearestinteger position. Using these rounded integer grid motion vectors as theoffset with respect to the current coding unit position in the tworeferences used for bi-prediction, a sum of absolute differences (SAD)between the blocks of coding unit size samples is computed. This sum ofabsolute differences is compared against a coding unit (CU)size-dependent threshold value and the rest or remaining of thedecoder-side motion vector refinement process steps is normativelyskipped when the sum of absolute differences is less than the thresholdvalue.

The appropriate pre-determined CU size-dependent threshold value isdetermined based on the desired average computational savings or energysaving traded-off against the drop in compression gains with a highthreshold.

FIG. 4 is a simplified flowchart of a method using decoder-sideverification or determination of an alignment level between motioncompensated patches according to an embodiment presented herein, wherethe conditional skipping of the remaining refinement process steps isbased on a rounded integer position-based cost function. The steps shownin the flowchart may be implemented as program codes or instructionsexecutable on one or more processors at the encoder side or the decoderside. The steps shown in the flowchart may be implemented usingelectronic components, digital and/or analog circuitry, logic elements,hardware, software, firmware, or combinations thereof. The method mayinclude, at step 401, a video decoder (video decoder and decoder areinterchangeably used herein) obtains unrounded merge motion vectorsbetween two motion compensated reference patches. A patch has apredetermined size of a current block of M×N pixels, M and N each arepositive integers. In some embodiments, M may be equal to N. In otherembodiments, M and N may be different. A patch may be a prediction unitor a coding unit and is referred to as a block of samples within apicture (frame). The unrounded merge motion vectors can be obtainedusing bilateral matching, template matching, or other techniques.Unrounded merge motion vectors can be merge motion vectors having aninteger pixel precision or a fractional pixel precision. The decoderrounds the unrounded merge motion vectors to obtain the rounded motionvectors at step 402. Rounding operation may include converting anunrounded motion vector to an integer pixel accuracy motion vector orreducing the pixel precision of the motion vector. For example, afractional precision motion vector can be rounded to an integer pixelprecision motion vector. The rounding operation may include the rightshifting of the vector value by one or more bits. In one embodiment, thevideo decoder may round the unrounded merge motion vectors to thenearest integer sample positions. In another embodiment, the videodecoder may round the unrounded merge motion vectors to the nearesthalf-pixel sample positions.

At step 403, the video decoder computes (calculates) a sum of absolutedifferences (SAD) between the two motion compensated reference patchesby performing simple motion compensation using the rounded motionvectors. The sum of differences SAD (first patch, second patch) in therespective reference pictures Ref0, Ref1 is a cost function fordetermining the best template match in the respective search spaces. Atstep 404, the decoder compares the SAD with a coding unit (CU)size-dependent threshold value to determine whether the SAD is less thanor not less than (i.e., equal to or greater than) the CU size-dependentthreshold value. When the SAD is determined to be less than the CUsize-dependent threshold value (“yes” in step 405), the decoder skipsthe decoder-side motion vector refinement (DMVR) process steps (step406) and performs the final motion compensation using the unroundedmerge motion vectors (step 408). When the SAD is determined to be notless than the CU size-dependent threshold value (“no” in step 405), thedecoder performs the decoder-side motion vector refinement (DMVR)process steps (step 407), and then performs the final motioncompensation (step 408). The decoder may refine the motion vectorcandidates based on a reference block in a search window that closelymatches the current block, i.e., the decoder may determine a newinterpolated motion vector (e.g., using bilinear interpolation) for thecurrent block when the SAD is not less than the CU size-dependentthreshold value, thereafter, the decoder then performs the final motioncompensation for the current block. In one embodiment, performing thefinal motion compensation may include rounding the unrounded mergemotion vectors to the nearest integer sample position in the horizontaldirection and in the vertical direction to obtain integer positionsamples, and performing an averaging operation on the integer positionsamples. In another embodiment, performing the final motion compensationmay include rounding the unrounded merge motion vectors to the nearesthalf-pixel sample position in the horizontal direction and in thevertical direction to obtain half-pixel position samples, and performingan averaging operation on the half-pixel position samples.

This embodiment is advantageous, among other advantages, in thatwhenever the metric evaluated is less than the threshold value, allothers aspects of decoder-side motion vector refinement such as bilinearinterpolation, mean value determination, and refinement cost functionevaluations at different positions, can be skipped and the finalnormative motion compensation can be performed. It should be noted thatin hardware implementations, it is possible in some embodiments tospeculatively perform bilinear interpolation in parallel with theevaluation of the metric for skipping the refinement to improve thetiming constraint (i.e., obtain more time or clock cycles to perform therefinement tasks).

Embodiment 2

In this embodiment, the decoder computes the sum of absolute differencesbetween the motion compensated patches in the two references involved inbi-prediction using a subset of samples from the bilinear motioncompensated interpolation performed using the sub-pixel accurate mergemotion vectors. This sum of absolute differences is then comparedagainst a pre-determined coding unit size dependent threshold value andthe rest of the refinement steps are normatively skipped when thedecoder determines that the sum of absolute differences is less than thethreshold value.

In one exemplary implementation, the bilinear motion compensatedinterpolation is performed in parallel (concurrently) on both thereference patches. In another embodiment, the production of theinterpolated samples across the two references are interleaved. In boththese cases, the sum of absolute difference calculation isadvantageously computed at the earliest possible time rather thanwaiting for all interpolated samples to be produced. This process alsoavoids loading the interpolated samples again. This reduces the hardwarelatency and improves the time available to perform the remainingrefinement computations required for the worst-case path.

In some implementations, a block of samples with the dimensions (size)equal to the coding unit dimensions are used for computing the sum ofabsolute differences. In such embodiments, the computation ofinterpolated samples required for the refinement positions other thanthe central position can be skipped when the sum of absolute differencesis determined to be less than the threshold value. In other embodiments,only a subset of interpolated samples are utilized for computing the sumof absolute differences. In these cases, the pre-determined thresholdvalue is modified to be a function of the number of samples used for thesum of absolute differences. When the decoder determines that the sum ofabsolute differences is less than the threshold value, the remaininginterpolation also can be skipped. In one embodiment, the subset ofinterpolated samples can be (coding_unit_width−2)*(coding_unit_height−2)samples in one embodiment. In another embodiment, the subset ofinterpolated samples can be(coding_unit_width*coding_unit_height−(coding_unit_width−2)*(coding_unit_height−2))samples in another embodiment, where coding_unit_width is a width of thecoding unit, and coding_unit_height is a height of the coding unit.

When bilinear interpolation constrains the bit depth of the interpolatedsamples, the pre-determined threshold is also adjusted accordingly byshifting down the threshold value derived at unconstrained bit-depth ofthe interpolated samples.

FIG. 5 shows a simplified flowchart of a method using decoder-sideverification, checking or determination of an alignment level betweenmotion compensated patches according to an embodiment presented herein,where the conditional skipping of the decoder-side motion vectorrefinement process steps is based on a subset of the bilinear motioncompensated interpolated samples. The steps shown in the flowchart maybe implemented as program codes or instructions executable on one ormore processors at the encoder side or the decoder side. The steps shownin the flowchart may also be implemented using electronic components,digital and/or analog circuitry, logic elements, hardware, software,firmware, or combinations thereof. The method may include, at step 501,a video decoder obtains unrounded merge motion vectors associated withtwo motion compensated reference patches. A patch has a predeterminedsize of a current block of M×N pixels, M and N each are positiveintegers. In some embodiments, M and N may be the same. In otherembodiments, M and N may be different. The unrounded merge motionvectors can be obtained using bilateral matching, template matching, orother techniques. The decoder also obtains motion compensatedinterpolated samples based on sub-pixel accurate merge motion vectors atstep 502.

At step 503, the decoder computes (calculates) a sum of absolutedifference (SAD) between the two motion compensated reference patchesusing a subset of the motion compensated interpolated samples. In someembodiments, the SAD can be computed in parallel or concurrentlyalongside the interpolation process. At step 504, the decoder comparesthe SAD with a coding unit (CU) size-dependent threshold value todetermine whether the SAD is less than or not less than the CUsize-dependent threshold value. In one embodiment, the CU size-dependentthreshold value is a function of the number of samples in the subset ofthe interpolated samples. When the SAD is determined to be less than theCU size-dependent threshold value (“yes” in step 505), the decoder skipsthe decoder-side motion vector refinement (DMVR) process steps (step506) and performs the final motion compensation using the unroundedmerge motion vectors (step 508). When the SAD is determined to be notless than (i.e., equal to or greater than) the CU size-dependentthreshold value (“no” in step 505), the decoder performs thedecoder-side motion vector refinement (DMVR) process steps using theunrounded merge motion vectors (step 507), and then performs the finalmotion compensation using the unrounded merge motion vectors (step 508).The decoder may refine the motion vector candidates based on a referenceblock in a search window that closely matches the current block, i.e.,the decoder may determine a new interpolated motion vector (e.g., usingbilinear interpolation) for the current block when the SAD is not lessthan the CU size-dependent threshold value, thereafter, the decoder thenperforms the final motion compensation for the current block using theunrounded merge motion vectors.

The second embodiment (embodiment 2) allows less coding gain decrease asa conservative threshold can still allow many coding units to skip theremaining steps of the refinement process. From a timing point of view,the sum of absolute difference operations required for the determinationor verification of the SAD against the threshold can be substantiallyhidden against the bilinear interpolation step and thus will not impactthe worst-case timing constraint.

Embodiment 3

In this embodiment, the early exit check (determination) is performedusing the mean-removed sum of absolute differences (MR-SAD) computed forthe center position where the refinement starts. In some embodiments,the center position's MR-SAD is evaluated first and compared against acoding unit size dependent threshold. If the MR-SAD for the centerposition is less than the threshold value, then the rest of therefinement process is skipped normatively.

In some embodiments, it is possible that MR-SAD evaluations at positionsother than the center position (e.g. certain positions within the firstiteration refinement points) may be evaluated alongside the MR-SADevaluation of the center position. However, these evaluated MR-SADs willhave no effect when the center position's MR-SAD is less than thepre-determined threshold value.

When interpolation constrains the bit depth of the interpolated samples,the pre-determined threshold value is also adjusted accordingly byshifting down the threshold value derived at the unconstrained bit-depthof the interpolated samples.

FIG. 6 shows a simplified flowchart of a method using decoder-sideverification, checking or determination of an alignment level betweenmotion compensated patches according to an embodiment presented herein,where the conditional skipping of the decoder-side motion vectorrefinement process steps is based on a mean-removed sum of absolutedifferences of the refinement start positon. The steps shown in theflowchart may be implemented as program codes or instructions executableon one or more processors at the encoder side or the decoder side. Thesteps shown in the flowchart may also be implemented using electroniccomponents, digital and/or analog circuitry, logic elements, hardware,software, firmware, or combinations thereof. The method may begin, atstep 602, a video decoder obtains motion compensated interpolatedsamples based on sub-pixel accurate merge motion vectors between twomotion compensated reference patches. At step 603, the decoder computes(calculates) a mean value of each of the patches for the center positionat which the refinement starts. The decoder also computes (at 603) amean-removed sum of absolute differences (MR-SAD) for the centerposition.

At 604, the decoder compares the MR-SAD against a coding unit (CU)size-dependent threshold value to determine whether the MR-SAD is lessthan or not less than (equal to or greater than) the CU size-dependentthreshold value. In one embodiment, the CU size-dependent thresholdvalue is a function of the bit depth of the sub-pixel accurate mergemotion vectors. When the MR-SAD is determined to be less than the CUsize-dependent threshold value (“yes” in step 605), the decoder skipsthe decoder-side motion vector refinement (DMVR) process steps (step606) and performs the final motion compensation using the merge motionvectors (step 608). When the MR-SAD is determined to be not less thanthe CU size-dependent threshold value (“no” in step 605), the decoderperforms the decoder-side motion vector refinement (DMVR) process stepsusing the merge motion vectors (step 607), and then performs the finalmotion compensation using the merge motion vectors (step 608). Thedecoder may refine the motion vector candidates based on a referenceblock in a search window that closely matches the current block, i.e.,the decoder may determine a new interpolated motion vector (e.g., usingbilinear interpolation) for the current block when the MR-SAD is notless than the CU size-dependent threshold value, thereafter, the decoderthen performs the final motion compensation for the current block usingthe merge motion vectors.

In some embodiments, computing the mean value of a patch may includeaveraging the luma intensity of all of the pixels within thecorresponding reference patch. The mean-removed sum of absolutedifferences (MR-SAD) may be computed using the following expression:MR-SAD=SAD(firstsample−mean(first_patch),second_sample−mean(second_patch))

Where first sample represents the first motion compensated interpolatedsample at the center of the first motion compensated reference patch,the second motion compensated interpolated sample at the center of thesecond motion compensated reference patch, mean(first block) representsan averaging operation of sample values of the first block, mean(secondblock) represents an averaging operation of sample values of the secondblock, the (first, second) patch may have a width and height of the(first, second) block in terms of samples, and SAD is a summingoperation of absolute differences of the values of samples of the firstpatch and the samples of the second patch.

This embodiment can ensure that no additional computations are incurredfor performing the conditional skipping of refinement check over what isrequired without such a check. Statistically, the computations of therefinement iterations are saved which can result in power savings andaverage cycle reduction in software.

The flowcharts shown and described above are intended to illustrateexemplary embodiments of determination, checking or verification of thealignment between motion compensated patches according to theembodiments presented herein. As one skilled in the art knows, thealignment checking and determination steps described herein may bere-arranged, modified, or combined to practice the present embodimentswithout departing from the scope of the present disclosure.

Embodiments in accordance with the present disclosure may be implementedin hardware circuitry, software program codes, computer instructionsexecutable by one or more processors or CPUs, or combination thereof.For example, embodiments can be one or more integrated circuits ordevices that perform the steps described herein. The program codes canbe executed on a digital signal processor (DSP). Embodiments can also beimplemented by one or microprocessors, one or more field programmablegate array (FPGAs).

Embodiments in accordance with the present disclosure provide anapparatus for determining the level of alignment between motioncompensated reference patches (coding units, blocks) against apredetermined CU size-dependent threshold value. The apparatus mayinclude circuitry configured to obtain unrounded merge motion vectorsbetween two motion compensated reference patches; round the unroundedmerge motion vectors to the nearest integer sample position to obtainrounded motion vectors; compute the sum of absolute differences (SAD)between the two motion compensated reference patches using the roundedmotion vectors; and determine whether the SAD is less than a CUsize-dependent threshold value. In the event that the SAD is less thanthe CU size-dependent threshold value, the circuitry of the apparatusskips remaining steps of a decoder-side motion vector refinement processand perform the final motion compensation using the unrounded mergemotion vectors. In the event that the SAD is not less than the CUsize-dependent threshold value, the circuitry of the apparatus performsthe remaining steps of a decoder-side motion vector refinement process,and then perform the final motion compensation using the unrounded mergemotion vectors.

In one embodiment, the circuitry of the apparatus derives the SAD usinga subset of motion compensated interpolated samples instead of using therounded motion vectors.

In one embodiment, the circuitry of the apparatus computes amean-removed sum of absolute differences (MR-SAD) for the integer pixelmotion search and compare the MR-SAD against the CU size-dependentthreshold value.

In some embodiments, the circuitry of the apparatus may include anintegrated semiconductor device or chip. In other embodiments, thecircuitry may include hardware components or device or a combination ofhardware and software, e.g., one or more processors, programmabledevices, or DSPs operable with instructions or program codes.

Following is an explanation of the applications of the encoding methodas well as the decoding method as shown in the above-mentionedembodiments, and a system using them.

FIG. 7 is a block diagram showing a content supply system 3100 forrealizing content distribution service. This content supply system 3100includes capture device 3102, terminal device 3106, and optionallyincludes display 3126. The capture device 3102 communicates with theterminal device 3106 over communication link 3104. The communicationlink may include the communication channel 13 described above. Thecommunication link 3104 includes but not limited to WIFI, Ethernet,Cable, wireless (3G/4G/5G), USB, or any kind of combination thereof, orthe like.

The capture device 3102 generates data, and may encode the data by theencoding method as shown in the above embodiments. Alternatively, thecapture device 3102 may distribute the data to a streaming server (notshown in the Figures), and the server encodes the data and transmits theencoded data to the terminal device 3106. The capture device 3102includes but not limited to camera, smart phone or Pad, computer orlaptop, video conference system, PDA, vehicle mounted device, or acombination of any of them, or the like. For example, the capture device3102 may include the source device 12 as described above. When the dataincludes video, the video encoder 20 included in the capture device 3102may actually perform video encoding processing. When the data includesaudio (i.e., voice), an audio encoder included in the capture device3102 may actually perform audio encoding processing. For some practicalscenarios, the capture device 3102 distributes the encoded video andaudio data by multiplexing them together. For other practical scenarios,for example in the video conference system, the encoded audio data andthe encoded video data are not multiplexed. Capture device 3102distributes the encoded audio data and the encoded video data to theterminal device 3106 separately.

In the content supply system 3100, the terminal device 310 receives andreproduces the encoded data. The terminal device 3106 could be a devicewith data receiving and recovering capability, such as smart phone orPad 3108, computer or laptop 3110, network video recorder (NVR)/digitalvideo recorder (DVR) 3112, TV 3114, set top box (STB) 3116, videoconference system 3118, video surveillance system 3120, personal digitalassistant (PDA) 3122, vehicle mounted device 3124, or a combination ofany of them, or the like capable of decoding the above-mentioned encodeddata. For example, the terminal device 3106 may include the destinationdevice 14 as described above. When the encoded data includes video, thevideo decoder 30 included in the terminal device is prioritized toperform video decoding. When the encoded data includes audio, an audiodecoder included in the terminal device is prioritized to perform audiodecoding processing.

For a terminal device with its display, for example, smart phone or Pad3108, computer or laptop 3110, network video recorder (NVR)/digitalvideo recorder (DVR) 3112, TV 3114, personal digital assistant (PDA)3122, or vehicle mounted device 3124, the terminal device can feed thedecoded data to its display. For a terminal device equipped with nodisplay, for example, STB 3116, video conference system 3118, or videosurveillance system 3120, an external display 3126 is contacted thereinto receive and show the decoded data.

When each device in this system performs encoding or decoding, thepicture encoding device or the picture decoding device, as shown in theabove-mentioned embodiments, can be used.

FIG. 8 is a diagram showing a structure of an example of the terminaldevice 3106. After the terminal device 3106 receives stream from thecapture device 3102, the protocol proceeding unit 3202 analyzes thetransmission protocol of the stream. The protocol includes but notlimited to Real Time Streaming Protocol (RTSP), Hyper Text TransferProtocol (HTTP), HTTP Live streaming protocol (HLS), MPEG-DASH,Real-time Transport protocol (RTP), Real Time Messaging Protocol (RTMP),or any kind of combination thereof, or the like.

After the protocol proceeding unit 3202 processes the stream, streamfile is generated. The file is outputted to a demultiplexing unit 3204.The demultiplexing unit 3204 can separate the multiplexed data into theencoded audio data and the encoded video data. As described above, forsome practical scenarios, for example in the video conference system,the encoded audio data and the encoded video data are not multiplexed.In this situation, the encoded data is transmitted to video decoder 3206and audio decoder 3208 without through the demultiplexing unit 3204.

Via the demultiplexing processing, video elementary stream (ES), audioES, and optionally subtitle are generated. The video decoder 3206, whichincludes the video decoder 30 as explained in the above mentionedembodiments, decodes the video ES by the decoding method as shown in theabove-mentioned embodiments to generate video frame, and feeds this datato the synchronous unit 3212. The audio decoder 3208, decodes the audioES to generate audio frame, and feeds this data to the synchronous unit3212. Alternatively, the video frame may store in a buffer (not shown inFIG. Y) before feeding it to the synchronous unit 3212. Similarly, theaudio frame may store in a buffer (not shown in FIG. Y) before feedingit to the synchronous unit 3212.

The synchronous unit 3212 synchronizes the video frame and the audioframe, and supplies the video/audio to a video/audio display 3214. Forexample, the synchronous unit 3212 synchronizes the presentation of thevideo and audio information. Information may code in the syntax usingtime stamps concerning the presentation of coded audio and visual dataand time stamps concerning the delivery of the data stream itself.

If subtitle is included in the stream, the subtitle decoder 3210 decodesthe subtitle, and synchronizes it with the video frame and the audioframe, and supplies the video/audio/subtitle to a video/audio/subtitledisplay 3216.

The present disclosure is not limited to the above-mentioned system, andeither the picture encoding device or the picture decoding device in theabove-mentioned embodiments can be incorporated into other system, forexample, a car system.

What is claimed is:
 1. A method of motion compensation for a block of aframe in a video with a decoder-side motion vector refinement (DMVR)process comprising: obtaining, by a decoder, motion compensatedinterpolated samples in two reference frames of the block based onsub-pixel accurate merge motion vectors of the block for the tworeference frames via a bilinear motion compensated interpolation;computing, by the decoder, a sum of absolute differences (SAD) betweentwo motion compensated reference patches using a subset of the motioncompensated interpolated samples; determining, by the decoder, whetherthe SAD is less than a coding unit (CU) size-dependent threshold value,the CU threshold value depending on a size of the CU; when the SAD isless than the CU size-dependent threshold value: skipping remainingsteps of the DMVR process for the block; and performing final motioncompensation using the sub-pixel accurate merge motion vectors of theblock to obtain reconstructed samples for the block; and when the SAD isnot less than the CU size-dependent threshold value: performing theremaining steps of the DMVR process to determine refined motion vectorsfor the block; and performing the final motion compensation using therefined motion vectors to obtain the reconstructed samples for theblock.
 2. The method of claim 1, wherein the subset of the motioncompensated interpolated samples are generated based on unrounded mergemotion vectors associated with the two motion compensated referencepatches.
 3. The method of claim 1, wherein the bilinear motioncompensated interpolation for generating the motion compensatedinterpolated samples in the two reference frames are performed inparallel.
 4. The method of claim 1, wherein the motion compensatedinterpolated samples are generated in an interleaved manner.
 5. Themethod of claim 1, wherein the subset of the motion compensatedinterpolated samples comprises a block of samples having a dimensionequal to a dimension of a coding unit (CU).
 6. The method of claim 5,wherein the subset of the motion compensated interpolated samplescomprises (CU_width−2)*(CU-height−2) samples, wherein CU_width is awidth of the coding unit and CU-height is a height of the coding unit.7. The method of claim 5, wherein the subset of the motion compensatedinterpolated samples comprises(CU_width*CU_height−(CU_width−2)*(CU-height−2)) samples, whereinCU_width is a width of the coding unit and CU-height is a height of thecoding unit.
 8. The method of claim 1, wherein the CU size-dependentthreshold value is a function of a number of samples in the subset ofmotion compensated interpolated samples.
 9. A decoder, performing motioncompensation for a block of a frame in a video with a decoder-sidemotion vector refinement (DMVR) process, the decoder comprising: one ormore processors; and a non-transitory computer-readable storage mediumcoupled to the processors and storing programming for execution by theprocessors, the programming, when executed by the processors,configuring the decoder to: obtain motion compensated interpolatedsamples in two reference frames of the block based on sub-pixel accuratemerge motion vectors of the block for the two reference frames via abilinear motion compensated interpolation; compute a sum of absolutedifferences (SAD) between two motion compensated reference patches usinga subset of the motion compensated interpolated samples; determinewhether the SAD is less than a coding unit (CU) size-dependent thresholdvalue, the CU threshold value depending on a size of the CU; when theSAD is less than the CU size-dependent threshold value: skip remainingsteps of the DMVR process for the block; and perform final motioncompensation using the sub-pixel accurate merge motion vectors of theblock to obtain reconstructed samples for the block; and when the SAD isnot less than the CU size-dependent threshold value: perform theremaining steps of the DMVR process to determine refined motion vectorsfor the block; and perform the final motion compensation using therefined motion vectors to obtain the reconstructed samples for theblock.
 10. The decoder of claim 9, wherein the subset of the motioncompensated interpolated samples are generated based on unrounded mergemotion vectors associated with the two motion compensated referencepatches.
 11. The decoder of claim 9, wherein the bilinear motioncompensated interpolation for generating the motion compensatedinterpolated samples in the two reference frames are performed inparallel.
 12. The decoder of claim 9, wherein the motion compensatedinterpolated samples are generated in an interleaved manner.
 13. Thedecoder of claim 9, wherein the subset of the motion compensatedinterpolated samples comprises a block of samples having a dimensionequal to a dimension of a coding unit (CU).
 14. The decoder of claim 13,wherein the subset of the motion compensated interpolated samplescomprises (CU_width−2)*(CU-height−2) samples, wherein CU_width is awidth of the coding unit and CU-height is a height of the coding unit.15. The decoder of claim 13, wherein the subset of the motioncompensated interpolated samples comprises(CU_width*CU_height−(CU_width−2)*(CU-height−2)) samples, whereinCU_width is a width of the coding unit and CU-height is a height of thecoding unit.
 16. The decoder of claim 9, wherein the CU size-dependentthreshold value is a function of a number of samples in the subset ofmotion compensated interpolated samples.
 17. A non-transitorycomputer-readable storage medium comprising instructions which, when theinstructions are executed by a computer, cause the computer to carry outmotion compensation for a block of a frame in a video with adecoder-side motion vector refinement (DMVR) process comprising:obtaining motion compensated interpolated samples in two referenceframes of the block based on sub-pixel accurate merge motion vectors ofthe block for the two reference frames via a bilinear motion compensatedinterpolation; computing a sum of absolute differences (SAD) between twomotion compensated reference patches using a subset of the motioncompensated interpolated samples; determining whether the SAD is lessthan a coding unit (CU) size-dependent threshold value, the CU thresholdvalue depending on a size of the CU; when the SAD is less than the CUsize-dependent threshold value: skipping remaining steps of the DMVRprocess for the block; and performing final motion compensation usingthe sub-pixel accurate merge motion vectors of the block to obtainreconstructed samples for the block; and when the SAD is not less thanthe CU size-dependent threshold value: performing the remaining steps ofthe DMVR process to determine refined motion vectors for the block; andperforming the final motion compensation using the refined motionvectors to obtain the reconstructed samples for the block.
 18. Thenon-transitory computer-readable storage medium of claim 17, wherein thesubset of the motion compensated interpolated samples are generatedbased on unrounded merge motion vectors associated with the two motioncompensated reference patches.
 19. The non-transitory computer-readablestorage medium of claim 17, wherein the bilinear motion compensatedinterpolation for generating the motion compensated interpolated samplesin the two reference frames are performed in parallel.
 20. Thenon-transitory computer-readable storage medium of claim 17, wherein themotion compensated interpolated samples are generated in an interleavedmanner.